Monte Carlo Simulations for Cost Benefit Analysis

A discussion

Zac Payne-Thompson

zachary.payne-thompson@dcms.gov.uk

The Department for Culture, Media and Sport

November 2, 2023

Pre-requisites

Brief History of CBA
Outline of current workflow in green book
What do we mean by sensitivities?

Analysis

Overview of distributions
Monte Carlo analysis
Interlude on Optimism Bias
Distributional approaches

Evaluation

What would this mean in practice?

Reflection

Does this matter?

A Brief History of CBA

Cost-Benefit Analysis, or CBA, is a powerful decision-making tool that quantifies the costs and benefits of various projects or policies. To understand its historical context, we need to go back to the 20th century, a period marked by significant changes in the role of central governments.

In the United States and elsewhere, central governments grew substantially during this time, and this growth played a crucial role in the evolution of CBA. It was in 1936, during the New Deal era in the United States, that CBA was first employed. Congress ordered government agencies to use CBA to evaluate projects related to flood control.

The use of CBA became increasingly popular among administrative agencies as the federal government continued to expand. But to truly appreciate the development of CBA, we must also consider the influence of Progressivism and the birth of welfare economics. These ideological shifts separated value-laden politics from the realm of administrative expertise grounded in scientific principles.

Historical Development of CBA

The New Deal Era

The New Deal government’s adoption of CBA in 1936 was a pivotal moment in its history, as it marked the first official use of CBA in a government context.

As the central government in the United States and other countries continued to grow, CBA gained rapid popularity among administrative agencies. This growth was not isolated; it was influenced by the rise of Progressivism during the late 19th and early 20th centuries. Progressives believed in the separation of politics, driven by values, from administrative decisions grounded in scientific principles.

This ideological shift paved the way for the development of modern welfare economics, which supplied the scientific principles necessary for the implementation of CBA. Early welfare economists believed that economic concepts could rationalise government policies, and their efforts gained further momentum in the 1950s and 1960s when governments sought technical assistance in developing formal CBA procedures.

Addressing the issues of Pareto Efficiency

The Hicks-Kaldor Compensation Principle

Definition

An allocative (i) change (ii) increases efficiency if the gainers from the change are (iii) capable of compensating the losers and still coming out ahead.

Each individual’s gain or loss is defined as the value of a hypothetical monetary compensation that would keep each individual (in his or her own judgement) indifferent to the change

Cost-benefit analysis examines whether policy changes satisfy the compensation principle or not

Addressing the issues of Pareto Efficiency

The Hicks-Kaldor Compensation Principle

CBA in Practice

CBA Popularity and Doubts in the 1960s and 1970s

In the 1960s, Cost-Benefit Analysis (CBA) gained popularity, even though there was no clear consensus on its theoretical foundation.
Government agencies and applied economists embraced CBA during this period.
However, by the 1970s, doubts started to emerge regarding the utility of CBA, both theoretically and practically.
These doubts were not just theoretical but also related to challenges and criticisms in applying CBA to real-world decision-making.

CBA in Practise

The Real Practice of CBA

While CBA is taught in textbooks with specific methodologies, its practical application in government agencies often differs.
Agencies may adapt CBA to their specific needs, using it as a tool to rationalize decisions made for various reasons, including political and administrative considerations.
It’s not uncommon for agencies to deviate from standard CBA procedures without always providing a clear rationale.
The actual practice of CBA can be influenced by external factors such as legal constraints, data availability, and practical limitations.

CBA in the Green Book

The Appraisal Process

Define the Problem
Establish Objectives
Identify Options
Appraise Options
Sensitivity Analysis
Decision Making
Implementation and Monitoring

Introduction - The Green Book is the UK Government’s guide for conducting appraisals of public sector projects and policies. - The appraisal process is designed to assess the economic, financial, social, and environmental impacts of proposed initiatives.

Key Steps in Appraisal

Define the Problem

Clearly articulate the problem that the policy or project intends to address.
Understand the underlying causes and set objectives.

Establish Objectives

Define specific, measurable, and achievable objectives for each option.
Objectives should cover economic, social, and environmental aspects.

Identify Options

Develop a range of possible options to address the problem.
Include a “do-nothing” option as a baseline for comparison.

Appraise Options

Conduct a rigorous assessment of each option, including a detailed cost-benefit analysis.
Consider the present value of costs and benefits over a specified timeframe.
Use appropriate discount rates to account for the time value of money.

Sensitivity Analysis

Test the robustness of the appraisal by varying key assumptions and parameters.
Assess how changes in inputs impact the results.

Decision Making

Compare the outcomes of the different options, considering not only financial but also social and environmental impacts.
Make informed decisions based on the appraisal results.

Implementation and Monitoring

Once a decision is made, implement the chosen option.
Establish a monitoring framework to track the actual performance against the expected outcomes.

Conclusion - The Green Book’s appraisal process ensures that public sector decisions are based on a comprehensive and systematic evaluation of options. - It promotes transparency, accountability, and evidence-based policymaking.

CBA in the Green Book

Sensitivity Analysis

Definition

Sensitivity analysis explores how the expected outcomes of an intervention are sensitive to variations in key input variables.
It helps understand the impact of changing assumptions on project feasibility and preferred options.

A key concept is the Switching Value: The value at which a key input variable would need to change to switch from a recommended option to another or for a proposal not to receive funding.

Identifying switching values is crucial to decision-making.

CBA in the Green Book

Sensitivity Analysis

Variable	Value
Site area	39 acre
Existing use land value estimate	£30,659 per acre
Future use land value estimate	£200,000 per acre
Land value uplift per acre	£169,341 per acre
Total land value uplift	£6.6m
Wider social benefits	£1.4m
Present Value Benefits (PVB)	£8m
Present Value Cost (PVC)	£10m
Benefit Cost Ratio (BCR = PVB / PVC)	0.8
Net Present Social Value (NPSV)	-£2m

Officials are appraising the treatment of a 39 acre contaminated land site, to be funded by a public sector grant. The remediation of the land would enable new businesses to move close to an existing cluster of businesses in a highly productive sector. The benefits of the intervention can be estimated by the change in the land value of the site (land value uplift). There is data on the current value and likely value of the land post remediation.

The total benefits are £8m when wider social benefits are added to the increase in land value as a result of the remediation. The costs of the remediation exceed the benefits so the BCR is less than 1 and the NPSV is negative. The switching value to turn the NPSV positive, so benefits outweigh costs, would be an approximate future land use value of £251,000 per acre equal to a land value uplift of approximately £221,000 per acre.

CBA in the Green Book

Optimism Bias

Definition

Optimism bias is the demonstrated systematic tendency for appraisers to be over-optimistic about key project parameters, including capital costs, operating costs, project duration and benefits delivery.

Adjust for optimism bias to provide a realistic assessment of project estimates.
Adjustments should align with risk avoidance and mitigation measures, with robust evidence required before reductions.
Apply optimism bias adjustments to operating and capital costs. Use confidence intervals for key input variables when typical bias measurements are unavailable.

CBA in the Green Book

Monte Carlo Analysis

Note

Monte Carlo analysis is a simulation-based risk modelling technique that produces expected values and confidence intervals. The outputs are the result of many simulations that model the collective impact of a number of uncertainties.

It is useful when there are a number of variables with significant uncertainties, which have known, or reasonably estimated, independent probability distributions.

It requires a well estimated model of the likely impacts of an intervention and expert professional input from an operational researcher, statistician, econometrician, or other experienced practitioner.

Monte Carlo Simulations for CBA

Data and Setup

project_id	low	central	high
1	64.37888	159.9989	223.8726
2	89.41526	133.2824	296.2359
3	70.44885	148.8613	260.1366
4	94.15087	195.4474	424.4830
5	97.02336	148.2902	471.6297
6	52.27782	189.0350	288.0247
7	76.40527	191.4438	236.4092
8	94.62095	160.8735	684.4839
…	…	…	…

Monte Carlo Simulations for CBA

Data and Setup

Objective: Create functions to generate different cost distributions based on user-specified parameters.
Process:
- Each function generates a sequence of possible project-level costs based on user-defined “high” and “low” values.
- Depending on the chosen distribution assumption, a probability distribution function is applied to create a vector of probabilities.
- The sample() function is used to randomly sample cost values from the sequence, with replacement, using the assumed probability distribution.

Monte Carlo Simulations for CBA

Data and Setup

Total Cost Distributions:
- These functions are applied to the project dataset to calculate total costs.
- The result is a vector of possible total project costs that can be plotted as a distribution.
- This approach allows for the exploration of different cost scenarios and provides a basis for risk analysis in project management.

Monte Carlo Simulations for CBA

1) Uniform Distribution

Project costs are modeled using a uniform distribution spanning low to high.

uniform_1 <- function(low, high){
  
}

Monte Carlo Simulations for CBA

1) Uniform Distribution

Project costs are modeled using a uniform distribution spanning low to high.

uniform_1 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
}

Monte Carlo Simulations for CBA

1) Uniform Distribution

Project costs are modeled using a uniform distribution spanning low to high.

uniform_1 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
  # Uniform Probability distribution function
  distribution <- dunif(sequence, min = low, max = high)
  
}

Monte Carlo Simulations for CBA

1) Uniform Distribution

Project costs are modeled using a uniform distribution spanning low to high.

uniform_1 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
  # Uniform Probability distribution function
  distribution <- dunif(sequence, min = low, max = high)
  
  # Sampling from possible costs using the assumed distribution function
  sample(x = sequence, size = 10000, replace = T, prob = distribution)
  
}

Monte Carlo Simulations for CBA

1) Uniform Distribution

Monte Carlo Simulations for CBA

1) Uniform Distribution

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

Project costs are modeled using a normal distribution with a mean defined as the midpoint between high and low, and a standard deviationthat is 1/4 of the distance between high and low.

This means that, if the data is truly normally distributed, then the low and high estimates represent the 95% confidence interval for an individual project’s cost.

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

This function looks like:

normal_2 <- function(low, high){
  
}

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

This function looks like:

normal_2 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)

}

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

This function looks like:

normal_2 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
  # Mean equal to the midpoint between low and high
  mean_x = (high-low)/2+low
  
}

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

This function looks like:

normal_2 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
  # Mean equal to the midpoint between low and high
  mean_x = (high-low)/2+low
  
  # Standard Deviation equal to 1/4 of the distance between low and high
  sd_x = (high-low)/4
  
}

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

This function looks like:

normal_2 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
  # Mean equal to the midpoint between low and high
  mean_x = (high-low)/2+low
  
  # Standard Deviation equal to 1/4 of the distance between low and high
  sd_x = (high-low)/4
  
  # Normal Probability Distribution Function
  distribution <- dnorm(sequence, mean = mean_x, sd = sd_x)
  
}

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)

This function looks like:

normal_2 <- function(low, high){
  
  # Set of possible costs
  sequence <- seq(from = 0, to = sum(data$high), by = 1)
  
  # Mean equal to the midpoint between low and high
  mean_x = (high-low)/2+low
  
  # Standard Deviation equal to 1/4 of the distance between low and high
  sd_x = (high-low)/4
  
  # Normal Probability Distribution Function
  distribution <- dnorm(sequence, mean = mean_x, sd = sd_x)
  
  # Sampling from possible costs using the assumed distribution function
  sample(x = sequence, size = 10000, replace = T, prob = distribution)
  
}

Monte Carlo Simulations for CBA

2) Normal Distribution (without a central estimate)